Method and apparatus for generating image using lidar

ABSTRACT

According to an aspect of embodiments, a method of generating an image by using LiDAR includes performing a reconstruction of a two-dimensional reflection intensity image, the performing of the reconstruction including projecting three-dimensional reflection intensity data that are measured by using the LiDAR as the two-dimensional reflection intensity image, and the method includes generating a color image by applying a projected two-dimensional reflection intensity image to a Fully Convolutional Network (FCN).

TECHNICAL FIELD

The present disclosure in some embodiments relates to a method andapparatus for generating an image using LiDAR.

BACKGROUND

The statements in this section merely provide background informationrelated to the present disclosure and do not necessarily constituteprior art.

Light Detection And Ranging (hereinafter, LiDAR) is used to measure thedistance to an object and obtain information reflected from the object.With LiDAR, the distance to the object is measured by using the timetaken to radiate light to an object and receive the light back, as wellas the amount of reflected light. Cameras used for imaging an objectwould produce unclear images due to their optical sensitivity to theinfluence of light or shadows, where LiDAR unaffected by light canprovide constant performance data regardless of the weather andillumination. For example, at nighttime imaging with a camera wouldproduce an image with hardly discernible objects or shapes, whereasLiDAR generates visually detectable and discernible data of objects evenat nighttime.

The LiDAR has been used to construct 3D Geographic Information System(GIS) information, and information measured by using LiDAR has beenvisualized by advanced technologies for applications in construction,aviation, defense and other fields. More recent technologies are underdevelopment for applying LiDAR to autonomous vehicles and mobile robots.

However, reflection intensity obtained using LiDAR is sparse, making itdifficult to identify or detect objects. This prompts LiDAR to involvecameras, when used to identify or detect objects.

DISCLOSURE Technical Problem

The present disclosure in some embodiments seeks to provide a method andan apparatus for generating an image using LiDAR.

Technical Problem

According to an aspect of embodiments, a method of generating an imageby using LiDAR includes performing a reconstruction of a two-dimensionalreflection intensity image, the performing of the reconstructionincluding projecting three-dimensional reflection intensity data thatare measured by using the LiDAR as the two-dimensional reflectionintensity image, and the method includes generating a color image byapplying a projected two-dimensional reflection intensity image to adeep learning network.

According to an aspect of embodiments, an apparatus for generating animage by using LiDAR includes a LiDAR projection image generation unitconfigured to reconstruct a two-dimensional reflection intensity imageby projecting three-dimensional reflection intensity data that aremeasured by using the LiDAR as the two-dimensional reflection intensityimage, and an image generation unit using a deep learning networkconfigured to generate a color image by applying a projectedtwo-dimensional reflection intensity image to a deep learning network.

SUMMARY

At least one aspect of the present disclosure provides a method ofgenerating an image by using LiDAR, the method including performing areconstruction of a two-dimensional reflection intensity image, theperforming of the reconstruction including projecting three-dimensionalreflection intensity data that are measured by using the LiDAR as thetwo-dimensional reflection intensity image, and the method includinggenerating a color image by applying a projected two-dimensionalreflection intensity image to a deep learning network.

Another aspect of the present disclosure provides an apparatus forgenerating an image by using LiDAR, which includes a LiDAR projectionimage generation unit and an image generation unit. The LiDAR projectionimage generation unit is configured to reconstruct a two-dimensionalreflection intensity image by projecting three-dimensional reflectionintensity data that are measured by using the LiDAR as thetwo-dimensional reflection intensity image. The image generation unit isconfigured to use a deep learning network for generating a color imageby applying a projected two-dimensional reflection intensity image tothe deep learning network.

Advantageous Effects

As described above, according to at least one embodiment, images asbrilliant as daylight can be obtained night and day, and as clear as thesun regardless of whether it's sunny or not. Applied to an autonomousvehicle in some embodiments, the present disclosure generates imagesless affected by the surrounding environment than when using a camera.Applied to crime prevention in some embodiments, the present disclosureprovides clear images even at night or on a cloudy day.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a configuration of an apparatus forgenerating an image using LiDAR according to at least one embodiment ofthe present disclosure.

FIG. 2 is a diagram sequentially showing an image generated according toat least one embodiment of the present disclosure.

FIG. 3 is a diagram of learning and reasoning processes performed by animage generation unit using a deep learning network according to atleast one embodiment of the present disclosure.

FIG. 4 is a diagram of a structure of a deep learning network accordingto at least one embodiment of the present disclosure.

FIG. 5 is a flowchart of a method of generating an image using LiDAR inaccordance with at least one embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, some embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings. In thefollowing description, like reference numerals designate like elements,although the elements are shown in different drawings. Further, in thefollowing description of some embodiments, a detailed description ofknown functions and configurations incorporated therein will be omittedfor the purpose of clarity and for brevity.

Additionally, various terms such as first, second, A, B, (a), (b), etc.,are used solely for the purpose of differentiating one component fromthe other, not to imply or suggest the substances, the order or sequenceof the components. Throughout this specification, when a part “includes”or “comprises” a component, the part is meant to further include othercomponents, not excluding thereof unless specifically stated to thecontrary. The terms such as “unit,” “module,” and the like refer tounits for processing at least one function or operation, which may beimplemented by hardware, software, or a combination of the two.

FIG. 1 is a block diagram of a configuration of an apparatus forgenerating an image using LiDAR according to at least one embodiment ofthe present disclosure.

The apparatus for generating an image using the LiDAR according to someembodiments of the present disclosure includes a LiDAR data acquisitionunit 110, a LiDAR projection image generation unit 120, and an imagegeneration unit 130 using a deep learning network.

The FIG. 1 illustrates a plurality of divided components, which may beintegrated into a single configuration, or a single component may bedivided into several component units.

The LiDAR data acquisition unit 110 utilizes LiDAR for measuring thedistance to the object and the amount of reflectance or reflectionintensity of light reflected from the object. Random presence or absenceof objects and the distance thereto will generate LiDAR data that is notonly generally inconstant, but also short of being dense. FIG. 2 showsat (a) an example visualization of data obtained by the LiDAR dataacquisition unit 110.

The RiDAR projection image generation unit 120 projects, intwo-dimensional coordinates, the reflection intensity data havingthree-dimensional coordinates obtained by the RiDAR data acquisitionunit 110. The reflection intensity data with the three-dimensionalcoordinates may have the form of a point cloud. At this time, theviewing angle, resolution, tilt angle, and height of the image to beprojected may be considered. A projection matrix may be used to projectthree-dimensional coordinates into two-dimensional coordinates. Forexample, Equation 1 as in the following shows a mathematical expressionfor transforming three-dimensional coordinates into two-dimensionalcoordinates by using a projection matrix. Here, X, Y and Z representthree-dimensional coordinates to be converted, and u and v representtransformed two-dimensional coordinates.

$\begin{matrix}{{s\begin{bmatrix}u \\v \\1\end{bmatrix}} = {\begin{pmatrix}f_{u} & 0 & c_{u} \\0 & f_{v} & c_{v} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{pmatrix}{\begin{pmatrix}R_{L}^{C} & t_{L}^{C} \\0 & 1\end{pmatrix}\begin{bmatrix}X \\Y \\Z \\1\end{bmatrix}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In addition, s is the scale factor, c_(u) and c_(v) are the principalpoint of the camera, f_(u) and f_(v) are the focal length, R_(L)^(C)∈R^(3×3) and t_(L) ^(C)∈R^(1×3) denote a rotation matrix and atransformation matrix, respectively, for transforming LiDAR into acamera position. FIG. 2 shows at (b) an example projection, intwo-dimensional coordinates, of the reflection intensity data havingthree-dimensional coordinates, obtained by the LiDAR data acquisitionunit 110.

The image generation unit 130 using the deep learning network generatesthe image by applying the data generated from the LiDAR projection imagegeneration unit 120, that is, the reflection intensity data projected intwo-dimensional coordinates, to the trained deep learning network. Theimage generated using the deep learning network may be a monochrome orcolor image. The image generation unit 130 using the deep learningnetwork will be described in detail below.

In order to facilitate the understanding of the present disclosure, eachof the components included in the image generating apparatus (FIG. 1)using the LiDAR according to some embodiments is divided into functionalcomponents, which in practice may be implemented as a single componentsuch as a CPU, MPU, GPU or ECU. Alternatively, those components may beimplemented through multiple devices. In particular, the imagegeneration unit 130 using the deep learning network of the FIG. 1 may beimplemented using a GPU. Specifically, although other processors mayrealize color image generation, the GPU is a good choice for processingthereof at a higher speed. Accordingly, the LiDAR projection imagegeneration unit 120 and the image generation unit 130 using the deeplearning network may be integrated with the LiDAR data acquisition unit110.

FIG. 3 is a diagram of learning and reasoning processes performed by animage generation unit using a deep learning network according to atleast one embodiment of the present disclosure.

Specifically, FIG. 3 is a diagram showing learning and reasoningprocesses for generating a color image.

First, in the learning process, a reflection intensity image 310projected in two-dimensional coordinates generated by the LiDARprojection image generation unit 120 is input to a deep learning network320 which then outputs color components 330. Selectively, threedimensional coordinates (or distance to an object) may be input to thedeep learning network 320 as well. The coefficients of deep learningnetwork 320 are trained so that color components 330 output from thedeep learning network 320 coincide with the color components of anoriginal color image 340. The original color image 340 used in thelearning process may be an image captured by a camera. In addition, theoriginal color image 340 may be an image whose shadow has been deletedfrom the image captured by the camera. In the present disclosure,learning is performed using shadowless images because LiDAR, unaffectedby shadows or light, can provide the same good data as daytime offers,even in an environment where shadows occur or even at night. In otherwords, the reflection intensity (or reflectance, reflectivity)information does not show the characteristics of a shadow, so thelearning is performed using the shadowless image. If learning involvedshaded images, color components could be distorted by shadows. Inaddition, obtaining shadowless images is a technology of significancefor autonomous driving algorithms for machines such as moving robots andautomobiles, and various studies thereof are under way.

When the deep learning network 320 utilizes the tanh function as anactivation function, the output value has a value between −1 and 1. Thiscauses a discrepancy that the color components extracted from theoriginal color image 340 have an effective range that do not coincidewith that of the color components 330 output from the deep learningnetwork 320, which needs a data range conversion 350 of the colorcomponents extracted from the original color image 340. Alternatively,the color components 330 output from the deep learning network 320 needa conversion to be within the valid range of the color componentsextracted from the original color image 340. The following descriptionillustrates the case of converting the range of the color componentsextracted from the original color image 340. Color components may berepresented by R, G, and B, and in general, their range of values isbetween 0 and 255 in an image. The color components are not necessarilyconverted to RGB, and they may be converted into various colorcomponents such as Gray, YUV, YCbYr and CIE Lab. However, since thecolor components 330 output by the deep learning network 320 has a valuebetween −1 and 1 by the activation function, the data range conversion350 is performed on the original color image 340 to have the range ofcolor components between −1 and 1. In other words, the range of data tobe converted varies in accordance with the active function of the deeplearning network.

In the following reasoning process, the reflection intensity imageprojection in the two-dimensional coordinates, which is generated by theLiDAR projection image generation unit 120 is input to the deep learningnetwork 320, to output the color components 330. As in the learningprocess, three-dimensional coordinates (or distances to objects) mayalso be selectively input to the deep learning network 320. The colorcomponents 330 output, having a value between −1 and 1, from the deeplearning network 320 undergo a data range conversion 370 to have a valuebetween 0 and 255. The converted value is used to generate a color image360.

FIG. 4 is a diagram of a structure of a deep learning network accordingto at least one embodiment of the present disclosure.

Specifically, FIG. 4 illustrates an image of 592×112 size as an example.Therefore, changes in image size will result in changes in the number ofconvolution groups or the number of sampling times.

The input to the deep learning network may be a reflection intensityimage projected in two dimensional coordinates. In this case, the inputof the reflection intensity alone constitutes one channel, and anadditional input of three-dimensional coordinates (or the distance tothe object) makes up two channels. The output of the deep learningnetwork may be three channels of R, G, B representing the components ofthe color image.

The deep learning network according to some embodiments may be composedof an encoding unit 410 and a decoding unit 420. The encoding unit unit410 and the decoding unit 420 of the deep learning network may be anasymmetrically configured Fully Convolutional Network (FCN).

The encoding unit 410 may include at least one convolution group and asub-sampling unit. The decoding unit 420 may also include at least oneconvolution group and an up-sampling unit. The convolution group may becomposed of at least one convolution block. The convolution block(convolution-K block) may be composed of a convolution layer including K3×3 filters, a batch normalization layer, and an active function, in theabove order. In addition, the convolution layer may be set to have astride of 1 and padding to be even, and the last convolution block ofthe decoding unit 420 may use tanh as an active function. The activefunction of all other convolution blocks may be Rectified Linear Unit(ReLU) (see Equation 2).

$\begin{matrix}{{{\tanh(x)} = {{2\left( \frac{1}{1 + e^{{- 2}x}} \right)} - 1}},} & {{Equation}\mspace{14mu} 2} \\{{{ReLU}(x)} = {\max\left( {0,x} \right)}} & \;\end{matrix}$

As shown in FIG. 4, the number of iterations of the convolution blockconstituting the i-th convolution group of the encoding unit 410 isN_(i) ^(e), and the number of iterations of the convolution blockconstituting the j-th convolution group of the decoding unit 420 isN_(j) ^(d), wherein the numbers of repetitions are variable.

The encoding unit 410 and the decoding unit 420 of the FCN may have anasymmetric structure by configuring the convolution blocks constitutingthe convolution group of the encoding unit 410 to have the total numberof Σ_(i) ⁶N_(i) ^(e), and the convolution blocks constituting theconvolution group of the decoding unit 420 to have the total number ofΣ_(j) ⁶N_(j) ^(d) so that Σ_(i) ⁶N_(i) ^(d)<Σ_(j) ⁶N_(j) ^(d), i.e., thetotal number of convolution blocks of the decoding unit 420 is largerthan that of the convolution blocks of the encoding unit 420.

The number of subsampling times of the encoding unit 410 may be a factorof 2 to which max-pooling may be applied. The number of up-samplingtimes of the decoding unit 420 may be a factor of 2 to which un-poolingmay be applied.

FIG. 5 is a flowchart of a method of generating an image using LiDAR inaccordance with at least one embodiment of the present disclosure.

The distance to the object and the reflection intensity are measured byusing LiDAR (in Step 510). For example, the distances to the object maybe expressed in three-dimensional coordinates by measuring distances onthe X, Y, and Z coordinates, respectively.

Three-dimensional reflection intensity data are generated by using thedistance and reflection intensity that are measured (520).

The generated three-dimensional reflection intensity data is projectedas a two-dimensional reflection intensity image (530). For example, thegenerated three-dimensional reflection intensity data may be convertedinto a two-dimensional reflection intensity image by using a projectionmatrix.

The projected two-dimensional reflection intensity image is applied to adeep learning network to generate a monochrome or color image (540). Thedeep learning network may be an FCN. At this time, in addition to theprojected two-dimensional reflection intensity image, the measureddistance or three-dimensional coordinates may be further input to theFCN. The FCN may be composed of an encoding unit and a decoding unit,and the encoding unit and the decoding unit may be configuredasymmetrically. The FCN may be trained by using a shadowless image as anoriginal image.

Although the steps 510 to 540 in FIG. 5 are described to be sequentiallyperformed, they merely instantiate the technical idea of someembodiments of the present disclosure. Therefore, a person havingordinary skill in the pertinent art could appreciate that variousmodifications, additions, and substitutions are possible by changing thesequences described in the respective drawings or by performing two ormore of the steps in parallel, without departing from the gist and thenature of the embodiments of the present disclosure, and hence the steps510 to 540 in FIG. 5 are not limited to the illustrated chronologicalsequences.

The steps shown in FIG. 5 can be implemented as computer-readable codeson a computer-readable recording medium. The computer-readable recordingmedium includes any type of recording device on which data that can beread by a computer system are recordable. Examples of thecomputer-readable recording medium are a magnetic recording medium(e.g., a ROM, a floppy disk, a hard disk, etc.), an optically readablemedium (e.g., a CD-ROM, a DVD, etc.), and the like, and a carrier wave(e.g., transmission through the Internet). Further, thecomputer-readable recording medium can be distributed in computersystems connected via a network, wherein computer-readable codes can bestored and executed in a distributed mode.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority to Patent Application No.10-2018-0100639, filed on Aug. 27, 2018 in Korea, the entire content ofwhich is hereby incorporated by reference.

1. A method of generating an image by using LiDAR, the methodcomprising: performing a reconstruction of a two-dimensional reflectionintensity image, comprising: projecting three-dimensional reflectionintensity data that are measured by using the LiDAR onto thetwo-dimensional reflection intensity image; and generating a color imageby applying a projected two-dimensional reflection intensity image to adeep learning network.
 2. The method of claim 1, wherein the deeplearning network is a Fully Convolutional Network (FCN).
 3. The methodof claim 1, wherein the deep learning network includes an encodingprocess and a decoding process, the encoding process and the decodingprocess being performed asymmetrically with respect to each other. 4.The method of claim 1, wherein the deep learning network comprises anetwork trained by using a shadowless image as an original image.
 5. Themethod of claim 1, wherein the generating of the color image by applyingthe projected two-dimensional reflection intensity image to the deeplearning network comprises: applying the projected two-dimensionalreflection intensity image and a measured distance to the deep learningnetwork to generate a color image.
 6. The method of claim 1, wherein theprojected two-dimensional reflection intensity image is represented by avalue between 0 and 1, or between −1 and 1, and wherein the color imageis represented by a value between 0 and
 255. 7. An apparatus forgenerating an image by using LiDAR, the apparatus comprising: a LiDARprojection image generation unit configured to reconstruct atwo-dimensional reflection intensity image by projectingthree-dimensional reflection intensity data that are measured by usingthe LiDAR onto the two-dimensional reflection intensity image; and animage generation unit using a deep learning network configured togenerate a color image by applying a projected two-dimensionalreflection intensity image to a deep learning network.
 8. The apparatusof claim 7, wherein the deep learning network is a Fully ConvolutionalNetwork (FCN).
 9. The apparatus of claim 7, wherein the deep learningnetwork includes an encoding unit and a decoding unit, the encoding unitand the decoding unit being configured asymmetrically with respect toeach other.
 10. The apparatus of claim 7, wherein the deep learningnetwork comprises a network trained by using a shadowless image as anoriginal image.
 11. The apparatus of claim 7, wherein the imagegeneration unit using the deep learning network is configured togenerate a color image by applying the projected two-dimensionalreflection intensity image and a measured distance to the deep learningnetwork.
 12. The apparatus of claim 8, wherein the projectedtwo-dimensional reflection intensity image is represented by a valuebetween 0 and 1, or between −1 and 1, and wherein the color image isrepresented by a value between 0 and 255.